Conversation
pg_mkdir_p creates each missing path component with a stat() followed by
mkdir(). If the stat() reports the component as absent but another process
creates it in the window before this process's mkdir(), mkdir() fails with
EEXIST and pg_mkdir_p treated that as a hard error -- unlike "mkdir -p", which
is meant to be idempotent and race-tolerant.
This shows up when several processes concurrently create paths that share an
ancestor directory: for example, parallel initdb runs whose data directories
live under a common temporary directory. One process wins the race to create
the shared ancestor and the others fail with
could not create directory "...": File exists
It is more easily hit on Windows, where stat() of a directory undergoing
concurrent creation can transiently fail, but the race exists everywhere.
After a failing mkdir(), accept the result when errno is EEXIST and the path
now exists as a directory; only then is the failure genuine.
A ctypes binding of libpq (bindings, constants, OIDs, library discovery, notification and result handling) plus a Session class providing synchronous, asynchronous, pipeline, COPY-free NOTIFY/notice and non-blocking query execution. This lets the Python test suite run SQL in-process without forking psql.
PostgresServer manages a cluster's lifecycle (initdb, start/stop/restart, promote), configuration, in-process SQL, log inspection, backup/streaming/ archiving/restore, WAL helpers, replication-slot helpers and connect_ok/ connect_fails connection assertions. PgBin runs client programs; the fixtures (pg_bin, create_pg, pg, conn, bindir, libdir) build the common test objects and tear them down automatically. Author: Jelte Fennema-Nio <postgres@jeltef.nl> Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
pgtap is a pytest plugin that emits TAP for the meson/prove harness when TESTLOGDIR is set, and maps a whole-module skip to success. The repository pyproject.toml carries the pytest configuration. meson gains a pytest feature option and a kind=='pytest' test branch, so each directory can list pytest suites beside its tap suites. Includes the suite's own self-tests. The root pyproject.toml also configures the suite's code quality gates: black for formatting, and pylint and mypy for linting and type checking (the dev-tooling dependency group lives in src/test/pytest/pyproject.toml; none of it is needed to run the tests). The whole suite is kept black-clean, pylint 10.00/10 and mypy-clean. Author: Jelte Fennema-Nio <postgres@jeltef.nl> Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Helpers used by the heavier test suites: a pg_regress runner, an OpenSSL-backed SSL server configurator, an slapd launcher, a stand-alone Kerberos KDC, and a launcher for the mock OAuth provider.
initdb, pg_ctl, pg_controldata, pg_resetwal, pg_config, pg_test_fsync, pg_test_timing, pg_archivecleanup, pg_waldump, pg_walsummary and scripts.
pg_basebackup, pg_rewind, pg_verifybackup, pg_combinebackup, pg_checksums, pg_amcheck and amcheck.
bin/pg_dump, bin/pg_upgrade and the test_pg_dump module. pg_upgrade delegates dump adjustment to a small Perl CLI wrapper (pyt/adjust_dump.pl).
auto_explain, basebackup_to_shell, bloom, dblink, oid2name, pg_prewarm, pg_stash_advice, pg_stat_statements, pg_visibility, test_decoding, vacuumlo, postgres_fdw and sepgsql.
Includes the test_checksums DataChecksums helper (pyt/conftest.py) and the injection-point-gated suites (test_aio, test_misc, xid_wraparound, ...).
ssl, ldap, ldap_password_func, kerberos, authentication, oauth_validator, libpq-oauth and ssl_passphrase_callback. These use the SSL/LDAP/Kerberos/OAuth helpers and skip cleanly where a daemon or build feature is unavailable.
interfaces/libpq, interfaces/ecpg/preproc, bin/psql (tab-completion and pager driven via pexpect), bin/pgbench, test/icu and tools/pg_bsd_indent.
Document the Python port of the Perl TAP suite: layout, how to run the tests under meson and directly with pytest, the shared fixtures, and the PostgresServer/Session/PgBin framework classes.
Enable the pytest suite (-Dpytest=enabled) on all jobs. This needs pytest installed where the images do not already provide it: via MacPorts on macOS (plus pexpect for the interactive psql tests, which need a pty and so skip on Windows), via pip on the Windows VS image, and via pacman on MinGW. The AddressSanitizer job needs one accommodation: the suite loads libpq in-process via ctypes, and dlopening an ASan-instrumented libpq into an uninstrumented python aborts because the ASan runtime must come first in the link order. Preload the ASan runtime for the test step to satisfy that; it is a no-op for the already-instrumented server binaries.
While iterating on the Python pytest port, run only the pytest tests: the Perl TAP tests double the run time and disk usage (the macOS runner has been hitting "No space left on device"). tap_tests and pytest are independent meson test kinds, so -Dtap_tests=disabled skips the Perl tests while -Dpytest=enabled keeps the Python ones. The Linux meson jobs configure with MESON_COMMON_PG_CONFIG_ARGS and otherwise let features auto-detect, so they do not pick up the setting from MESON_COMMON_FEATURES; pass -Dtap_tests=disabled explicitly in their meson setup commands. This is a temporary CI-only change to be reverted before the work is finalized.
Extract the SQLSTATE from a failed result (PQresultErrorField, which bindings.py already declares) onto ResultData.sqlstate, carry it on QueryError, and add named QueryError subclasses (QueryCanceled, UniqueViolation, DeadlockDetected, ...) so a test can write `with pytest.raises(QueryCanceled):` instead of catching the generic QueryError and string-matching its message. query_safe raises the SQLSTATE-specific subclass via query_error_for(); every subclass remains catchable as QueryError / LibpqError, and an unmapped SQLSTATE still raises a plain QueryError.
|
Andrew — I diffed my pytest port [1] against pytap/v3 to find capabilities mine has that yours doesn't, with an eye to handing back anything worth taking. Our two framework trees have disjoint git history (both built independently on upstream master), so a whole-tree diff is mostly noise; instead I went capability by capability. Here's the honest accounting. Worth taking — opened as its own PR:
Already covered in pytap/v3 — no PR needed (noting them so you know I checked):
Stylistic deltas — take them or leave them, not worth a PR each:
That's the complete set — the rest of the divergence is the two frameworks' different execution models (psql-by-default + opt-in libpq vs. in-process libpq throughout) and formatting, neither of which is a "feature mine has that yours lacks." Happy to rework #7 in whatever shape fits, or close it if you'd rather not carry it. |
libpq: SQLSTATE-based error matching for query failures
Owner
Author
|
@gburd I have merged your PR I'm happy to make stylistic changes - my python-fu is relatively limited. I prefer my simpler one method execution model, seems cleaner to me, although possibly less consistent with how we do things now (which I regard as a crock). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.